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Abstract 

Let Go be a connected, simply connected real simple Lie group. Suppose that Go has a 
compact Cartan subgroup To, so it has discrete series representations. Relative to To there is 
a distinguished positive root system A + for which there is a unique noncompact simple root 
v, the "Borel - de Siebenthal system" . There is a lot of fascinating geometry associated to 
the corresponding "Borel - de Siebenthal discrete series" representations of Go- In this paper 
we explore some of those geometric aspects and we work out the i^o-spectra of the Borel - 
de Siebenthal discrete series representations. This has already been carried out in detail for 
the case where the associated symmetric space Gq/Kq is of hermitian type, i.e. where v has 
coefficient 1 in the maximal root so we assume that the group Go is not of hermitian type, 
in other words that v has coefficient 2 in /i. 

Several authors have studied the case where Gq/Kq is a quaternionic symmetric space and 
the inducing holomorphic vector bundle is a line bundle. That is the case where fi is orthogonal 
to the compact simple roots and the inducing representation is 1-dimensional. 



1 Introduction 

One of Harish-Chandra's great achievements was the existence theorem for discrete series repre- 
sentations of a semisimple Lie group. He characterized the groups with discrete series represen- 
tations by the equal rank condition, he found the explicit formulae on the regular elliptic set for 
the characters of the discrete series, and he showed that those formulae specify the characters. 
At the same time (and as a main motivation) he was able to explicitly construct a particularly 
simple series, the holomorphic discrete series, for those groups where the corresponding Rieman- 
nian symmetric space is a bounded symmetric domain in a complex Euclidean space. For the 
other discrete series representations however, the actual construction has remained less explicit, 
although there are several beautiful realizations. 

In this paper we initiate the study of a certain family, the so-called Borel - de Siebenthal 
discrete series, from a point of view as close as possible to that of Harish-Chandra for the 
holomorphic discrete series. This is motivated in part by the work of Gross and Wallach for 
the scalar case of the quaternionic discrete scries GW . As in that case we obtain in particular 
the admissibility of the series for a small subgroup of the maximal compact subgroup. At 
the same time we discover a rather appealing geometry for the coadjoint orbit that one wants 
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to attach to the discrete series in question. In particular we give a detailed classification of 
the possible structures of such orbits in terms of explicit prehomogeneous vector spaces with 
relative invariants. We feel these deserve attention in their own right; and while we do give the 
construction for the Borel - de Siebenthal discrete series here, including the explicit iio-types 
and the (important) admissibility for a small subgroup K\ of we defer further analysis of 
continuations of the series to a sequel to this paper. In particular, we shall then elucidate the role 
of the relative invariants in constructing rather singular representations in the continuation of 
the series. We mention that much of this has been carried out in the quaternion line bundle case 
in [GWj . We also mention the papers |Kn2] and |Kn3j treating such questions for the indefinite 
orthogonal and symplectic groups; here methods from [GWj are used, and the connection to 
the continuation of unitary modules in the sense of Vogan (with criteria for good and fair range 
of unitarity of cohomologically induced modules) is made clear. Our approach seeks to employ 
analytic methods and the geometry of the orbits, and in particular to use reproducing kernels, 
see e.g. |WaWj . 

Several questions concerning discrete series representations may be resolved by our methods, 
for example the question of finding admissible branching laws, where one obtains direct sum 
decompositions with finite multiplicities. By applying admissibility of K\ such results may be 
obtained in complete analogy with what happens for holomorphic discrete series representations. 

In this paper we give a complete description of the geometry of the elliptic coadjoint orbits 
corresponding to the Borel - de Siebenthal discrete series. They are open Go-orbits in cer- 
tain complex flag manifolds and we give precise results on their maximal compact subvarieties 
(which are compact hermitian symmetric spaces) and the holomorphic normal bundles to those 
subvarieties. We use this structural information to give a concrete geometric construction of 
the representations in this series, including the structure of the Kg-types. Our construction of 
the -Ko~types provides an analogue of the i"Co~type decomposition of holomorphic discrete series 
representations. 

The quaternionic discrete series, studied by Gross and Wallach |GWj in the line bundle 
case, is the special case of the Borel - de Siebenthal discrete series, where the maximal root is 
compact and is orthogonal to all the other compact positive roots, or equivalently (see |W0j ) 
where Kq has a local direct factor isomorphic to Sp(l). While each complex simple Lie group G 
has exactly one noncompact real form Go that has quaternionic discrete series representations, 
every real simple Lie group Go with rank Go = rank K o has either holomorphic discrete series 
representations or Borel - de Siebenthal discrete series representations. Thus the Borel - de 
Siebenthal discrete series is the natural extension of the holomorphic discrete series. 

Our geometric approach allows us to extend several results from [GW . While Gross and 
Wallach constructed quaternionic discrete series representations on spaces of holomorphic forms 
with values in a line bundle, we also allow vector bundles - which is natural in the more general 
setting considered here. Our construction also provides good concrete examples of minimal 
cohomology degree realizations of discrete series representations in the sense of Kostant [Ko3j . 

Our basic tool is complex differential geometry and the associated cohomology groups. An 
important component of this is a collection of basic spectral sequence arguments, already implicit 
in the paper [ST]. See also [S2j, [S3] and |W4j . Here we make use of some technical results of 
M. Eastwood and the second named author from |EWj for some crucial identifications of duals 
of finite dimensional representations of reductive Lie groups, in particular for keeping track of 
the action of the center in terms of the highest weights and the Dynkin diagrams. 

Our results include a careful collection of the data attached to the orbits in question, and an 
explicit formula for the ifo _ types in the Harish-Chandra module corresponding to the discrete 
series in question. As a by-product we find two natural sets of strongly orthogonal roots, one 
corresponding to the hermitian symmetric space Kq/Lq and the other corresponding to the 
riemannian symmetric space Gq/Kq. They fit together to realize the orbit Gq(zq) = Gq/Lq as a 
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kind of Sicgel domain of Type II. This should provide useful coordinates for explicit calculations 
of the elements in the Harish-Chandra module. 

In Section[2]we work out the general structure of the complex manifold D = Gq(zq) = Go /To- 
We describe the action of lo on the tangent and normal spaces to the maximal compact subvariety 

Y = Kq(zq) = Kq/Lq, and on their duals, then we discuss a negativity condition that is crucial 
to the realization of our discrete series representations. 

In Section [3] we list all instances of simple Lie algebras Qq corresponding to Borel - de 
Siebenthal root orders. Setting aside the well-understood hcrmitian symmetric cases, we work 
out the precise structure of the algebras 60 and to and the parts of the complexified Lie algebra 
g that correspond to the holomorphic tangent space of Y and the holomorphic normal space of 

Y in D, including the representations of lo on those two spaces. In each case this allows explicit 
parameterization of the Borel - de Siebenthal discrete series. 

In Section S] we consider the prehomogeneous space (L, Ui) where ui C g represents the 
holomorphic normal space to Y in D. There we describe the algebra of relative invariants, 
using our knowledge of the representation of L on the symmetric algebra S(ui). In most cases 
we can be explicit, but in some we must rely on general results of Sato and Kimura [SK . 
These invariants are (in addition to being interesting in themselves) relevant for the next step of 
understanding the analytic continuation of the discrete series; here the ring of regular functions 
on the zero set of an invariant will correspond to a module in this continuation. We intend to 
follow this idea in a sequel to this paper. 

In Section [5] we assemble our preparations and work out the exact if-spectrum of the Borel 
- de Siebenthal discrete series representations. Our main result here, which is the main result 
of the paper, is Theorem l5.23l In a final example we look a the "sufficient negativity" condition 
that ensures the non- vanishing of exactly the right analytic cohomology group, and we compare 
it to the corresponding condition for individual if-types - this will indicate a possibility of 
continuing the discrete series family. 

2 Notation and the Basic Fibration 

In general we use capital Latin letters for Lie groups with subscript for real groups and no 
subscript for complexifications. We use the corresponding small Gothic letters for Lie algebras, 
again with subscript for real Lie algebras and no subscript for complexifications. Our basic 
objects are a connected simply connected simple real Lie group Go, its Lie algebra Qo, the 
complexification G of Go, and the Lie algebra g of G. Here G is a connected simply connected 
complex Lie group and the inclusion go t— * defines a homomorphism Go — > G with discrete 
central kernel. 

When we omit a subscript where there had been one before, we mean complexification. 

Fix a Cartan involution 9 of Go and go- The fixed point set Kq = G is a maximal compactly 
embedded subgroup of Go- As usual, we decompose g = 60 + So and q = t + s into (±1)- 
eigenspaces of 6, where to (resp. 6) is the Lie algebra of K (resp. K). 

We now make two assumptions: 

(2.1) rank Go = rankiio > and the symmetric space So : = Go/ -Ko is not of hermitian type. 

In particular Ko is a maximal compact subgroup of Go . Both Go and Kq are simply connected 
semisimple groups with finite center. 

Fix a maximal torus To C Kq. Then To is a compact Cartan subgroup of Go, and a celebrated 
theorem of Harish-Chandra says that Go has discrete series representations. 



3 



The construction of Borel and de Siebenthal [BoS] provides a positive root system A + = Aq 
for (g, t) such that the associated simple root system ^> = contains just one noncompact 
root. We denote 

(2.2) if? = {ipi, . . . ,tpt} (Bourbaki root order) and v e ^ is the noncompact simple root. 

Every root a G A + has expression a = ^ni(a)^>j. Since we have excluded the hermitian 
case, the coefficient of v in the maximal root \i is 2. Further, a root is compact just when the 
coefficient of v in its expansion is or ±2, noncompact just when that coefficient is ±1. Also, 
('J/ \ \y\) U {— fi} is a simple root system for (6, t). Grading by the coefficient n„ of v we have a 
parabolic subalgebra of g given by 

(2.3) q = [ + u_, reductive part [ = t + g Q and nilradical u_ = u_2 + u_i 

n„=0 

where Ui = J2 n =j S Q - The opposite parabolic is q opp = [ + u + where u + = Ui + U2. Note that 

(2.4) s = u_i + Ui, so q n s = u_i, and t = u_2 + I + U2. 

On the group level, we have the parabolic subgroup Q C G where Q has Lie algebra q. 
The group Q has Chevalley semidirect product decomposition LU- where L is the reductive 
component and U- is the unipotent radical. Note that Go n Q is a real form L of L and that 
Lq is the centralizer in Kq of a circle subgroup of To- The parabolic Q defines a complex flag 
manifold Z = G/Q, say with base point zq = 1Q, and and open orbit D = Gq(zo) = Gq/Lq. 
The complex manifold D has maximal compact subvariety Y = Kq(zq) = Kq/Lq, which is a 
smaller complex flag manifold K/(K n Q). 

Our choice of signs in (|2.3p is such that 

u+ is the holomorphic tangent space of D at zq , 

(2.5) U2 is the holomorphic tangent space of Y at zo , and 
Ui is the holomorphic normal space of Y in D at zq . 

Since Go /Kq is irreducible but not hermitian we know that the action of to on Sq is absolutely 
irreducible. Thus the action of t on s is irreducible. From |W2, Theorem 8.13.3] we know that 
the action of [ on each u, is irreducible. It will be convenient to have the notation 

(2.6) Tj : representation of L on the vector space Uj. 

The contragredient (dual) of Tj is r* = r_j. Some obvious highest or lowest weight spaces of 
the Tj are given by 

^ t 2 has highest weight space g M and r_ 2 has lowest weight space g_ M , 

T\ has lowest weight space g„ and r_i has highest weight space g_„ . 

Note that the degree degr^ = dime ik. If i ^ it is the number of roots a such that n^a) = i. 
The basic tool in this paper is the real analytic fibration 

(2.8) D — > Sq with fiber Y, in other words Gq/Lq —* Go/Ko with fiber Kq/Lq. 

The structure of the holomorphic tangent bundle and the holomorphic normal bundle to Y 
in D is given by (|2.5[) . (|2.6|) and (|2.7p . In the next section we will make this explicit. The 
fibration (|2.8p was first considered by W. Schmid in [ST] and |S2j for a related situation in 
which L = T , and then somewhat later by R. O. Wells and one of us [WeWj without that 
restriction. A much more general setting, which drops the compactness assumption on Lq, is 
that of the double fibration transform (see |FHWj and the references there), where So is replaced 
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by a complcxification S C G/K. Specialization of the double fibration transform to the Borel - 
de Siebenthal setting is carried out in [EWj . 

The simple root system ^ = {ipi, . . . , ipt} of g defines the system 
(2-9) S = {&,...,&} where fj^f = S id 

of fundamental simple weights. Let 7 be the highest weight of an irreducible representation of 
Lq. For our discussion of the Borel - de Siebenthal discrete series in Section [5] we will need to 
know exactly when (7 + p g , a) < for all positive complementary roots a (roots a that are not 
roots of [) where p g denotes half the sum of the positive roots of g. The condition is Theorem 
EE2 below. 

Define v* by (v* = for tpj 7^ v and 2{v*, v) = (u, v) (i.e. the fundamental weight dual 
to v). Then 7 6 it^ decomposes as 

(2.10) 7 = 70 + tv* where (70, v) = and t G E. 

Define A, : = {a e Aq \ n v (a) — i}, so Uj = 5Z QeA . B" f° r * e l^-*-' ±2}. Thus the positive root 
system decomposes as A + = (Ao (~l A + ) U Ai U A2. The highest weight of T2, representation 
of [ on U2 = J2a Sa is /i- If we subtract a positive combination of roots of \& \ {v} from /1 we 
decrease the inner product with 7 + p g . Thus 

(2.11a) (7 + p B ,a) < for all a 6 A 2 if and only if (7 + p B ,p) < 0. 

The highest weight of r_i is — z^, so n has highest weight w[*(^) where w® is the longest element 
of the Weyl group of L Thus 

(2.11b) (7 + p s ,a) < for all a € A% if and only if (7 + p B ,u;°(i/)) < 0. 

As 1/* is orthogonal to the roots of [ it is fixed by the inverse of w®, so (is* , w?(u)) = (y* , v) = 1. 
Using the decomposition (|2.10p . and combining (|2.11ap and (|2. 1 1£>[) . we have 

Theorem 2.12 The following conditions are equivalent. 

1. The inequality (7 + p B ,a) < holds for every root a G Ai U A2 (i.e. every positive 
complementary root) 

2. Both t < -5(70 + /°8>m) an d t < -(70 + p B , W^(f)}. 

Remark 2.13 In Theorem 12.121 it is automatic that (7 + p g ,(3) > for every positive root 
of t, so the conditions of Theorem 12.121 ensure that (7 + p B ,a) ^ for every root a, in other 
words that 7 + p g is the Harish-Chandra parameter of a discrete series representation of Go- 
Specifically, in our setting, the conditions of Theorem 12.121 will characterize the Borel - de 
Siebenthal discrete series. 

3 Classification 

In this section we give a complete list the simple Lie algebras go for which the hypotheses (|2.1j) 
hold. We then specify the complex parabolic subalgebra q C g and the real subalgebras 60 and 
Co of go- Next, we give precise descriptions of the representations Tj and their representation 
spaces Ui. Much of this is done using the Dynkin diagrams to indicate highest weights of 
representations. There the special cases, where Gq/Kq is a quaternionic symmetric space, are 
visible at a glance: they are the ones where —p connects directly to v in the extended Dynkin 
diagram of g. 

We will denote highest weights of representations as follows. In Dynkin diagrams with two 
root lengths we denote short root nodes by black dots • and long roots by the usual circles °. 
Extended diagrams are those with the negative of the maximal root p attached by the usual 
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rules. Recall the system S of fundamental simple weights from (|2.9[) . The (irreducible finite 
dimensional) representation of G and g of highest weight n^i is indicated by the Dynkin 
diagram of g with m written next to the i th node, except that we omit writing zeroes. So for 
example the adjoint representations are indicated by 





2 or 1 1 
o Ui O ■ • • o 


B e ,£^3 


1 
1 








2 

• • ■ • • m o 






G 2 


l 


F 4 


1 


E 6 


O c 

( 


3 O O 

,1 


E 7 


o o o o o o 

1 

o 


E s 


O c 

c 


5 O 

1 

> 







We will use this notation for 6 as well. It can be identified from its simple root system 
\Pf = (\P \ {is}) U {— /i}. We'll follow the notation of [BE] , except that we won't darken the dots. 
Thus the diagram of I consists of the diagram of g, except that the o (resp. •) at the node for 
the noncompact simple root v is replaced by an x (resp. Klj^- In the diagram of I, a symbol x 
or E3 indicates the 1-dimensional center of L. The irreducible representation of L with highest 
weight ^2 rii^i now indicated by the Dynkin diagram of I with rii written next to the i th node, 
for rii 7^ 0, whether that node is o, •, x or H. 

If v* is the fundamental simple weight corresponding to the noncompact simple root and 
x G t by a(x) = (£, a) for a£t*, then lo has center f&Lv* . 

Now we use the fact that Kq is connected, simply connected and semisimple. The simple 
root system ^{ decomposes into 

'■ connected component that contains — /i, and 
: the complement of in ^ 

This results in decompositions Kq = K± x K 2 and Lq = L\ x L 2 , which we make explicit in 
each case. 

In the following we list the Dynkin diagrams, with the possibilities of the noncompact simple 
root v among the simple roots fa. Also in the picture one finds the extended Dynkin diagram 
node for —fi where fi is the maximal root. Diagrams of Type A do not occur because v must 
have coefficient 2 in the expression of n as a linear combination of simple roots. We now consider 
the cases where g is of type B. 



3.1 Case 5pm (4, 2£-3). Here G is the 2-sheeted cover of the group 50(4, 2£ - 3) which is 
a real analytic subgroup of the complex simply connected group Spin(2£ + 1; C). Its extended 
Dynkin diagram is 



(3.1a) 



fa 



v 

-o- 

•'fa 



fa- 



fa 



(type Bi , £ > 2) 



Thus 6 is 



$1 $3 



--P- 



and [ is ^T" 



Of course one can also look at Lq as a subgroup of Ko, and from that viewpoint the diagram of [ is obtained from 
that of £ on replacing the o at the node for —fj, with a x . However it is more convenient to look at Lo as a subgroup 
of Go, and the diagram of [ from that viewpoint, when we consider the action of L on the subspaces Ui. 
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Now the decompositions to = 61 ® £2 and k> = h® b are 

(3.16) 6 =sp(l)8(sp(l)eso(2^-3)) and Iq = iSLu* ® (sp(l) xso(2^-3)) 

where 1/* the fundamental simple weight corresponding to v. The representation of t on s 

ol o . 

has highest weight — v = —ifa so its diagram is Q i 1 ■ Using (|2.7p . the 

representation 



(3.1c) r 2 : [ on U2 is ° f ° 

Also, the action r_i of I on u_i is J _2 1 c , so the dualizing diagram 

method of [EWj shows that the representation 



(3. Id) ti : I on Ui is j _\ \ 

Here dimu 2 = 1 and dimui = (21 - 3)(2£ - 4). 

3.2 Case Spin(2p, 2£-2p+l), 2<p <£. Here G is the 2-sheeted cover of SO{2p, 2£-2p+l), 
2 < p < I, contained in Spin(2l + 1; C). Its extended Dynkin diagram is 



(type B e ,£>3) 



(3.2a) o y 
/i 

^1/^2" rfp-l ffip+l fyi—lfyl q •■■ —_o— V— Q — ^}=» 

Thus Ms </ ' ' and [ is y)ip2 ipp-i Wp+i fye-iipe 
~ M 

Now the decompositions to = 61 © ?2 and to = ti © fa are 

(3.26) ! = so(2p)®so(2£-2p+l) and [ = u{p)®so(2£~2p+l) = iWLv* ®su(p)(Bso{2£~2p+l). 



o — -p ■ -o o- 

The representation of t on s has highest weight —v = —ip p : <f 
Using (|2.7|) . the representation 



(3.2c) T2 : Ion U2 is 



Also, the action r_i of [ on u_i is u u " l~— 3 i L so the dualizing 

diagram method of [EWj shows that the representation 

-x- 



(3. 2d) ti : [ on Ui is \ _\ 1 

Here dimii2 = p(p — l)/2 and dimui = p(2£ — 2p+ 1). 

3.3 Case Spin(A, 1). Here Go is the (universal) double cover of the group SO(4, 1). Its 
extended Dynkin diagram is 

::0 (type B 2 ) 



(3.3a) ipi ip2 
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Thus t is —fi and [ is . Now the decompositions to — £\ © 62 and to = Ii © [2 are 

(3.36) 6 = sp(l) ffisp(l) and to = ©Sp(l). 

o o 

Here 6 acts on s with highest weight — v = — ip2- 1 1 ■ Using (|2.7p . the representation 

o. K 

(3.3c) r 2 : I on U2 is —2 . 

Q H 

Also, the action r_i of [ on u_i is 1 —2 so the dualizing diagram method of [EWj shows 
that the representation 



(3.3d) ri : [ on Ui is 1 — 1 . 

dimu 2 = 1 and dimui = 2. 

3.4 Case Spin(2£, 1), I > 1. Here G is the universal (2-sheeted) cover of the group SO(2£, 1) 
with £ > 1. Its extended Dynkin diagram is 

(type Be, t > 2) 



(3.4a) _ o; ; 

Thus Ms Y and I is 

Now the decompositions t = 61 © ?2 and [ = ti © [2 are 

(3.46) to = so(2£) and [ = u(£) = iRv* © su(£). 



The representation of t on s has highest weight — v = —tpf. </ 1 . Using 

(|2.7p . the representation 



(3.4c) r 2 : [on U2 is 



Also, the action r_i of I on u_i is 1 —2 so the dualizing diagram method of 

[EWJ shows that the representation 



(3.4d) T\ : [on Ui is 1 

Here diniU2 = £{£ — l)/2 and dimui = £. This exhausts the cases where g is of type B, and we 
go on to consider the cases where g is of type C. 

3.5 Case Sp(p,£ — p),l < p < i. Here Go is simply connected, and its extended Dynkin 
diagram is 



(3.5a) -/i Vi V>2 ip P ipt-i i>e 



(type d,£>l) 



Thus t is - < /T^i $2 $ P -i ^£^£ with = {— //, i/> 2 , • • • , V>p-l} and 

*t 2 = {V'p+i ; V'p+2, ■ ■ ■ ,^}, and [ is $1 $2 8 ^-1 ^ ■ 
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Now the decompositions to = 61 © £2 and k> = h® 1% are 

(3.56) t = Sp(p) ®sp(£ -p) and [ = u(p) ©sp(^-p) = iM^* © su(p) © sp(£ - p). 



The representation of 6 on s has highest weight — 2/ = — i/j p : 
Using (|2.7[) . the representation 



(3.5c) t 2 : Ion U2 is 2 



The action r_x of [ on u_i is 1—21 i so the dualizing diagram 

method of [EWj shows that the representation 



(3.5d) ti : [ on Ui is \ _\ \ 

Here dimu2 = (p — l)(p + 2)/2 and dimui = 2p(£ — p). 

3.6 Case Sp(l,t — 1). Here Go is simply connected, and its extended Dynkin diagram is 



v 



(3.6a) -n -01 ^2 i>t-i ipi 



(type Q , £ > 1) 



Thus 6 is —fi $2 fyt—i tyt and I is $2 ^£-1 D ^ 

Now the decompositions 60 = ti © ?2 and to = Ii © [2 are 

(3.66) 6 =sp(l) ®sp(£- 1) and [ = ®sp(£- 1). 



The representation of 6 on s has highest weight — u = —ipi: 1 
Using (|2.7p . the representation 



(3.6c) r 2 : [ on U2 is 2 



The action r_i of [ on u_i is _2 ] , so the dualizing diagram method of 

[EWJ shows that the representation 



(3.6d) ti : [on ui is 



Here dimu2 = 1 and dimui = 2£. This exhausts the cases where g is of type C, and we go on 
to consider the cases where g is of type D. 

3.7 Case Spm(4, 11 — 4),£ > 4. Here G is the 2-sheeted cover of 50(4, 2£ - 4) contained in 
Spin(2£; C). Its extended Dynkin diagram is 



v 

-o- 




(type D e ,£>4) 



ipi 

(3.7a) o-' 

Tims Ms ^ ~$£*^>1>t an d [is ^ x ^ 



Now the decompositions 60 = Ii © £2 and [q = [1 © h are 



(3.76) t = sp(l) © (fip(l) © so(2^ - 4)) and [ = iRv* © (fip(l) © so{2£ - 4)) . 
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The representation of 6 on s has highest weight — v = —-02 so its diagram is i 1 
Using (|2.7[) . the representation 



(3.7c) 



r 2 : Ion 112 is f 



Also, the action r_i of I on u_i is J _o 1~ 
method of [EWj shows that the representation 



, so the dualizing diagram 



(3.7d) t\\ ton Ui is \ _^ 1 

Here dimu2 = 1 and dimui = 4(£ — 2). 



3.8 Case £0(4, 4). Here Go is the 2-sheeted cover of the group SO{4, 4) that is contained in 
Spin(8; C). Its extended Dynkin diagram is 



(3.8a) 



■01 s'fy 




"03 
04 



(type D 4 \ 



-/'- 



>03 



Thus E is t 1 t 3 and [ is fa * 

-fj, 04 

Now the decompositions 60 = £1 © 62 and [0 = h © h are 
(3.86) t = -sp(l) © (sp(l) © sp(l) © sp(l)) and [ = 



(SP(1)©SP(1)©5P(1)) 



ol ol 

The representation of i on s has highest weight — v = —02 so its diagram is 
Using (|2.7p . the representation 



T "1 



(3.8c) 



Also, the action r_i of [ on U-i is 
shows that the representation 



T2 : ton U2 is \ 
^o l 

lo K: 



2~\^ , so the dualizing diagram method of |EWj 



(3.8<2) 

Here dimii2 = 1 and dimui = 



, . lo X< 

n : I on Ui is _p 



3.9 Case Spin(2p,2£ - 2p),2 < p < £ - 2. Here G is the 2-sheeted cover of the group 
SO(2p,2£ — 2p) that is contained in Spin(2£;C), with 2 < p < £ — 2. Its extended Dynkin 
diagram is 

o p a ( type De ' ^ > 5 ) 

(3.9a) _ -- ^° ^ 

0«-i 



Thus t is " ^0^.0, and [ is 0T?r " ^t^+Tl^^ 
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Now the decompositions to =%i® h and k> = h® b are 

(3.96) t = so(2p) © so{2£ - 2p) and t = u(p) © so(2^ - 2p) = © su(p) © so(2£ - 2p). 



The representation of i on 5 has highest weight —v = —ip p - J J~ 

Using (12. 7p . the representation 



(3.9c) 



T2 : Ion U2 is 



Also, the action r_i of [ on u_i is 
diagram method of |EWj shows that the representation 



■o — x — o- 

1 -2 1 



so the dualizing 



(3.9<2) 



n : I on Ui is 



-X — °— 

-1 1 



Here dimii2 = pip — l)/2 and dimui = 2p(£ — p). This exhausts the cases where g is of type D, 
and thus exhausts the classical cases. We go on the the exceptional cases. 

3.10 Case G2,a 1 a 1 - Here Go is the split real Lie group of type Gi. It has maximal compact 
subgroup 50(4). Its extended Dynkin diagram is 

v 



Thus 6 is ^ 1 ^ and [ is • 
Now the decompositions to = t\ © ?2 and to = Ii ffi b are 

(3.106) to =sp(l) ffisp(l) and [ = iRv* ffisp(l). 

• o . . 

The representation of t on s has highest weight — v = ~xp2- 3 1 ■ Using (|2.7j) . the 
representation 



(3.10c) r 2 : [on u 2 is * f 

Also, the action r_i of [ on u_i is "3 _?2 so t ne representation 

(3.10d) t x : [onuiis * ^ . 

Note that Ti|rjn has degree 4, is self-dual, and has an antisymmetric bilinear invariant. Also, 
dimu2 = 1 and T2|[[.i] is trivial, so that bilinear invariant is given by the Lie algebra product 
Ui x m — ► U 2 . 

3.11 Case F^AiCg ■ Here Go is the simply connected real Lie group of type F4 whose maximal 
compact subgroup has 2-sheeted cover Sp(l) x Sp(3). Its extended Dynkin diagram is 



O" 



(3.11a) -/j, ip t ip 2 ^3 fa 



(Type F 4 ) 
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Thus t is —fjt ip2 ^4 and [ is x ^2 ^3 $"4 • 

Now the decompositions 60 = i\ © ?2 and = i\ ffi h, are 

(3.116) t = sp(l)ffisp(3) and [ = iRu* ©sp(3). 



The representation of t on s has highest weight — v = —ipi. l l . Using (|2.7I) . 

the representation 



x- 

(3.11c) t 2 : Ion U2 is 1 



Also, the action r_i of I on u_i is —2 1 so the representation 



x- 

(3. lid) n : ton Ui is 1 1 

Note that ti|[( n has degree 14, is self-dual, and has an antisymmetric bilinear invariant. Also, 
dimu2 = 1 and T2|[[j] is trivial, so that bilinear invariant is given by the Lie algebra product 
Ui x Ui — > u 2 . 

3.12 Case F4b 4 . Here Go is the simply connected real Lie group of type F4 with maximal 
compact subgroup Spin(9). Its extended Dynkin diagram is 

v 

(3.12a) £ £ Js" ^4 (Type Fi) 

-[1 Vl ^2 ^3 IpX 4>2 1p3 

Thus 6 is o o n . an d [ i s o □ . m . 

Now the decompositions 60 = £1 ffi ?2 and to = ti ffi la are 
(3.126) 6 = so(9) and [ = iRis* © so(7). 

The representation of t on 5 has highest weight — v = — tp^; o o c ■ . Using (|2.7[) . 

the representation 



o » H 

(3.12c) t 2 : I on U2 is 1 

1 -2 

Also, the action r_i of [ on u_i is o ° » h so the representation 



(3.12d) n : I on Ui is 1 — 1 

Note that has degree 8, is self-dual, and has a symmetric bilinear invariant. In effect T\ 

is the action of Spin(7) on the Cayley numbers, and r 2 is its action (factored through SO(7) on 
the pure imaginary Cayley numbers, thus dimii2 = 7 and dimui = 8. 
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3.13 Case E^a x A 5 ,i- Here Go is the group of type Eq whose maximal compact subgroup is 
the 2-sheeted cover of SU(2) x SU(6). The noncompact simple root v = 03, so L is of type 
T\A\Ai. We do not consider the case v — 05 separately because the two differ only by an outer 
automorphism of Eq. Here the extended Dynkin diagram (Bourbaki root order) is 



05 



01 03 
o o — 



0. 



(3.13a) 




06 
— O 



-/<< 

■■o 



(Type E 6 ) 



05 06 

Thus t is o < 2 -fi 



01 

and [ is ° — 



0, 



05 06 



'02 



Now the decompositions t = Si © ?2 and I = li © h are 

(3.136) 6 = su(6) ffi su(2) and [ = (su(5) iKi/*) © su(2). 

The representation of t on s 



1 

has highest weight —v = —ip3: ° 



Using (|2.7p . the representation 



(3.13c) 



T2 : 1 on U2 is 



Also, the action r_i of I on u_i is » 



(3.13d) 



Ti : I on Ui is ° 




so the representation 



Note that Ti\un has degree 20 and is not self-dual. We have dimu2 = 5 and dimui = 20 



3.14 Case Eq^AiAs^- Here Go is the group of type Eq with maximal compact subgroup 
SU{2) x SU(6). The noncompact simple root v — 2 , so L is of type TiA 5 . The extended 
Dynkin diagram (Bourbaki root order) is 



05 



01 03 
O O — 



(3.14a) 




— o 



02 
V 



-/'- 

■•o 



05 06 

rpi , . 01 03 4^ ° , r . 01 03 % 

thus t is o o cr and I is ° o— 

H 1 

o 

Now the decompositions t = Si © 62 and [q = li © b are 
(3.146) 6 = sp(l) © su(6) and [ = 



(Type E 6 ) 

05 06 



©su(6). 



The representation of t on s has highest weight — v = —02: 
(|2.7|) . the representation 



Using 



(3.14c) 



t 2 : I on u 2 is 
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Also, the action r_i of I on U-i is 



-2 

-x 



so the representation 



(3.14d) 



Ti : I on Ui is 




Note that has degree 20, is self-dual, and has an antisymmetric bilinear invariant. Also, 

dimii2 = 1 and T2|[[.t] is trivial, so that bilinear invariant is given by the Lie algebra product 
Ui x Ui — > u 2 . In brief, dimu 2 = 1 and dimui = 20. 



3.15 Case ^7,Ai0 6 ,1- H ere Go is the group of type Ej with maximal compact subgroup that 
is the 2-sheeted cover of ST/ (2) x Spin(12). The noncompact simple root v = so L is of 
type T\Dq and the extended Dynkin diagram is 



(3.15a) 



6^2 



(Type E r ) 



Thus t is and [ is 

A ^2 

Now the decompositions to = ti © £2 and = Ii © [ 2 are 
(3.156) t = sp(l) © so(12) and [ = iRv* so(12) 



The representation of t on s has highest weight — v 
(|2.7p . the representation 



1 



Using 



(3.15c) 



r 2 : I on u 2 is 



Also by (|2.7p . the representation r_x of [ on u_i is 
representation 



-2 1 



so the 



-1 



(3.15d) 



ti : I on Ui is 



Note that has degree 32, is self-dual, and has an antisymmetric bilinear invariant. Also, 

dimu 2 = 1 and t"2 | rt,t] is trivial, so that bilinear invariant is given by the Lie algebra product 
Ui x Ui — > u 2 . In brief, dimu 2 = 1 and dimui = 32. 



3.16 Case Ei_A 1 D fi ,2- Again Go is the group of type £7 with maximal compact subgroup that 
is a 2-sheeted cover of SU(2) x Spin(Y2), but now noncompact simple root v = ipe, so L is of 
type T1A1D5 and the extended Dynkin diagram is 

v 



-/'- 



(Type E 7 ) 



(3.16a) 



6^2 



14 



o o o o o 



Thus t is 



"A* Wl ^3 



O Q <>- O- 



and I is 

A ip 2 



V>4 i>5 4>7 
6 i>2 



Now the decompositions to = ti © 62 and to = li © b are 
(3.166) t = so(12)ffisp(l) and [ = (so (2) © so(10)) © fip(l) = (iRv* ffiso(lO)) ffisp(l). 



The representation of 6 on s has highest weight —v = —ipe'- 
(|2.7p . the representation 



1 1 



Using 



(3.16c) 



T2 : Ion U2 is 



Also by (|2.7p . the representation r_i of [ on u_i is 
representation 



1 -2 1 



so the 



(3.16d) 



Ti : I on Ui is 



-o 9 o x o 

-1 1 



61 



Note that Ti\un has degree 16 and is not self-dual. In brief, dimu2 = 10 and dimui = 16. 

3.17 Case £7,^7 ■ Here Go is the group of type £7 with maximal compact subgroup SU(8)/ {±1}. 
The noncompact simple root v — ip2 , so L is of type TiEq and the extended Dynkin diagram is 



(3.17a) 



~Wa $~5 ^6 $7 



(Type £7) 



v 6tp 2 



Vl ^3 



1p4 Vb Wl 



Thus t is —fx yji $4 $5 yjg yj 7 and [ is 

Now the decompositions B = £1 © h and io = h® h are 
(3.176) t = su(8) and [ = (u(l) © u(7)) n su(8) = iRis* © su(7). 



The representation of 6 on s has highest weight — v = —tp2' 
Using (|2.7[) . the representation 



(3.17c) 



T2 : Ion U2 is 



Also by (|2.7p . the representation r_x of [ on u_i is 
representation 



so the 



(3.17d) 



T\ : 1 on Ui is 



x-1 
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Note that Ti\un has degree 35 and is not self-dual. In brief, dimii2 = 7 and dimui = 35 



3.18 Case Es^ 8 . Here Go is the group of type Eg with maximal compact subgroup locally 
isomorphic to Spin(16). The noncompact simple root v — ipi, so L is of type T1D7 and the 
extended Dynkin diagram is 



(3.18a) 



~4h $5 $6 



0^2 



(Type E s ) 



03 



04~~0l 0^6 $7 $8 "A* , >< ^3 % % $8 

and I is 

A02 



Thus 6 is 

Now the decompositions 60 = £1 ffi ?2 and to = Ii ffi b are' 
(3.186) « = 50(16) and [ = so(2) © so(14) 



©so (14). 



The representation of t on s has highest weight —v = —ipi- 
Using (|2.7p . the representation 



(3.18c) 



T2 : Ion U2 is 



Also by (|2.7[) . the representation r_i of [ on u_i is 
representation 



x o- 

-2 1 



so the 



(3.18d) 



t\ : [on ui is 



Note that T\\un has degree 64 and is not self-dual; dimu2 = 14 and dimui = 64. 

3.19 Case E^a-lEt- Here Go is the group of type Es with maximal compact subgroup that 
has ST/ (2) x E? as a double cover.. The noncompact simple root v = so L is of type T1E7 
and the extended Dynkin diagram is 

v 



^01 $3 



(3.19a) 



~4h ^5 $6 $7" 



0^2 



^ (Type E s ) 



Thus 6 is 



■01 03 



04 05 06 07 
i -02 



, . . 01 03 

and [ is 



Now the decompositions to = £1 ffi 62 and to = li ffi [2 are 
(3.196) t = Sp(l) © e 7 and [ = iM^* © e7. 



4 05 06 07 
A 02 



The representation of t on s has highest weight —v = —ips,'- 
Using (|2.7p . the representation 



-o o 

1 1 



-X 
1 



(3.19c) 



72 : [on U2 is 



16 



Also by (|2.7[) . the representation r_i of [ on u_i is 
representation 



-x 

so the 



(3.19c?) T\ : [ on ui is 



1 -1 



Note that Ti|r[ n, has degree 56, is self-dual, and has an antisymmetric bilinear invariant. Also, 
dimu2 = 1 and T2|ri n is trivial, so that bilinear invariant is given by the Lie algebra product 
Ui x m — ► U 2 . 

This completes our run through the exceptional cases. 



4 Prehomogeneity and Relative Invariants for (L,Ui) 

Consider a connected linear algebraic group with a rational representation on a complex vector 
space. We say that the triple consisting of the group, the representation and the vector space 
is prehomogeneous if there is a Zariski-dense orbit. When no confusion is possible we omit the 
representation. A general theorem of Vinberg on graded Lie algebras (see |Knll Theorem 10.19]) 
shows that (L,Ui) is prehomogeneous. Or one can verify that fact by running through the lists 
of Section [3] and the classification of |SK| . In fact we will do the latter in order to describe the 
algebra of relative-invariant polynomials on Ui and the L-orbit structure of Ui for each instance 
of (L,Ui). We shall also use the notation V for the fundamental space Ui and V* = u_i for its 
dual space. 

We recall some material on prehomogeneous spaces as it applies to (L,Ui). There is no 
nonconstant L-invariant rational function / : ui — » C because the L-invariance would force it 
to be constant on the Zariski-dense L-orbit [SKI Proposition 3 in §2]. By relative invariant 
for (L, Ui) we mean a nonconstant polynomial function / : ui — > C such that f(££) = x(£)f(0 
for some rational character x '■ L — > C x . The quotient of two relative invariants with the same 
character would be an L-invariant rational function of Ui, hence constant, so a relative invariant 
/ is determined up to scalar multiple by its character x [SKL Proposition 3 in §4] . In particular 
all the f c : £ i— > /(c£) are proportional, so / is a homogeneous polynomial. It will be convenient 
to denote 

(4.1) A(L,Ui) : the associative algebra of all relative invariants of (L,Ui). 



The regular set for (L,Ui) is the open L-orbit Oq := Ad(L)£o C Ui and the singular set 
is its complement Ui \ Oq. Let V\, . . . , V e be those components of the singular set that are of 
codimcnsion 1 in ui. For each i, V$ is the zero set of an irreducible polynomial /j. The algebra 
of relative invariants for (L, m) is the polynomial algebra C[/i, . . . , f e ] |SK| Proposition 5 in §4]. 
In particular (L,Ui) has a relative invariant if and only if the its singular set has a component 
of codimension 1. So far we haven't used irreducibility of L on Ui, but now we use it to see |SK[ 
Proposition 12 in §4] that e ^ 1, i.e. that either A(L, Ui) = C (in other words (L.Ui) has no 
relative invariant) or ,A(L,Ui) has form C[/]. 

4.2. Cases SO(2p,r). We first consider the various cases where Go is the universal covering 
group of the indefinite orthogonal group SO(2p,2q) or SO(2p,2q + 1). For convenience we 
write that as SO(2p,r). Then L consists of all (g °) where a is in the image of the standard 



embedding i : U(p) ^ SO{2p) and where b e SO(r). Here s = {( ? x *)| x £ R 2 P xr } 
and the (conjugation) action ( g ° ) G L on So is given by x i— > axb' 1 . Now m = C pxr with the 
action of L ~ GL(p; C) x SO(r, C) given by £ = ( ^ ° ) : z ^ £ x z£^ . Then f(z) := det (z 



t. 
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transforms by f(£(z)) = det (£) 2 f(z). However it is a relative invariant only when it is not 
identically zero, i.e. when p 5; r. 

On the other hand, if p > r then the (SL(p; C) x SO(r; C))-orbit of ( ^ r \ is open in C pxr , 

so there is no nonconstant (SL(p;C) x SO(r; C))-invariant. It follows that there is no relative 
invariant for L. 

Summary: if p ^ r then A(L, Ui) = C[/] where f(z) := det (z ■ t z), polynomial of degree 2p. 
If p > r then A{L, Ui) = C. 

4.3. Cases Sp(p,q). We next consider the cases where Go = Sp(p,q). Then L consists of 
all ( o ° ) where a is in the image of the standard embedding t : U(p) '—* Sp(p) and where 
b £ Sp(q). Here S = {( °. *) | x £ HP*?} = W x 9 and the (conjugation) action (g °) 6 L on 
So is given by x i— ► axb' 1 . Now Ui = C px2q with the action of L ~ GL(p;C) x Sp(q;C) given 

by ^ = ( o 1 ) : z 4-^2 Let J = f _° g q" J , so Sp(q; C) is characterized by b ■ J ■ l b = J. 
Let Pf denote the Pfaffian polynomial on the space of antisymmetric 2q x 2q matrices, so 
Pf (m) 2 = det(m). Then f (z) := Pf (z ■ J ■ t z) transforms by f(£z) = det (t)f(z). And of 
course / is a relative invariant only when it is not identically zero. For that we must have the 
possibility that the p x p antisymmetric matrix z ■ J ■ i z is nonsingular, which is the case just 
when both that p ^ 2q and p is even. 

As before, if p > 2q then the (SL{p;C) x Sp(q; C))-orbit of ( ^ 3 ) is open in C px2 «, so 
there is no nonconstant (SL(p; C) x Sp(q; C)) -invariant, and thus no relative invariant for L. 

If p ^ 2q but p is odd we define m £ C px2? by m^j = 1 if i ^ p and i is odd, mj^+j = 1 if i ^ p 
and i is even, all other entries zero. The point is that the row space of m has dimension p and 
has nullity 1 relative to the bilinear form J that defines Sp(q; C). Then (SL(p; C) x Sp(q; C))(m) 
consists of all elements of C px2q whose row space has dimension p and nullity 1 relative to J, 
and that is open in C px29 . As above, it follows that there is no relative invariant for L. 

Summary: if p ^ 2q and p is even then A(L, Ui) = C[/] where f(z) := Pf (z ■ J ■ t z), 
polynomial of degree p. If p > 2q of if p is odd then A(L, Ui) = C. 

We have completely described A(Lui) when G is classical. Except for a few extreme cases, 
the representation t\ \li of the derived group L 1 = [L, L] failed to be self dual because of a tensor 

factor i ° ° . In many of the exceptional group cases, t\\li is self-dual, so it 

has a bilinear invariant, and when that bilinear invariant is symmetric it generates A(L,\\\). 
Also, if that bilinear invariant is antisymmetric, then ti(L') is contained in the symplectic group 
J of the bilinear invariant, and if a is the representation of J on Ui then S 2 (S 2 (a)) contains 
the trivial representation with multiplicity 1, and that gives a quartic invariant that generates 
A(L, Ui). Before formalizing these statements we look back at Section [3] to see the first five 
columns of 



Case 


Go 


degn 


self dual? 


bilinear invariant 


relative invariant 


(13.101) 


Gi^AxAx 


4 


yes 


antisymmetric 


degree 4 


(13.111) 




14 


yes 


antisymmetric 


degree 4 


(13.121) 




8 


yes 


symmetric 


degree 2 


(13.131) 


E&^AxA^.l 


20 


no 


none 


none 


(13.141) 


Ee,A!A 5 .2 


20 


yes 


antisymmetric 


degree 4 


(13.151) 


E7,A 1 D 6 ,1 


32 


yes 


antisymmetric 


degree 4 


(13.161) 


El,A x D & ,2 


16 


no 


none 


none 


(13.171) 


E 7 ,A7 


35 


no 


none 


degree 7 


(13.181) 


Eg,,D 8 


64 


no 


none 


degree 8 


(13.191) 


Es,A!E- 


56 


yes 


antisymmetric 


degree 4 



The information of the last column of Table [33] is contained in )SK| . but we can give a short 
direct proof of the cases where there is a relative invariant, as follows. 
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Lemma 4.5 If T\\l> is self-dual there are two possibilities. Either it has a nonzero symmetric 
bilinear invariant b and A(L, Ui) ~ C[b], or it has a nonzero antisymmetric bilinear invariant and 
A(L,ui) = C[f] where f has degree 4. In the non self-dual case (|3.17j) we have A(L, Ux) = 
where f has degree 7, and in the non self-dual case (|3.18J) we have A(L, Ui) = C[/] where f has 
degree 8. 

Proof. If the bilinear invariant b is symmetric, then since it has degree 2 it must generate 
A(L, Ui). If b is antisymmetric, then in each of the five relevant cases of Table [44] we compute 
symmetric powers S 2 {t\\l>), S 3 (ti\l') and <S 4 (ti|l') to see that we first encounter a T\{L')- 
invariant in degree 4. (This degree 4 semiinvariant can also be seen by a classification free 
argument [Pj Proposition 1.4].) 

Consider the two non self dual cases of Table WM for which we claim a t\ |^ -invariant . In 
case p.!7|) we compute the S t (t\\i,i s ) for 2 ^ r ^ 7 to see that we first encounter a T\{L')- 
invariant in degree 7, and and in case (|3.18|) we compute the S r (Ti\r / >) for 2 ^ r ^ 8 to see that 
we first encounter a ri(L')-invariant in degree 8. □ 

5 Negativity and Kq— types 

In this section we discuss negativity of a homogeneous holomorphic vector bundle over Gq/Lq 
and the -Ko _ types of the resulting discrete series representations. 

Recall some notation from Section O The flag domain D = Gq(zq) = Gq/Lq is an open Go- 
orbit in the complex flag manifold Z = G/Q, where zq = 1Q is the base point and Lq = GqHQ. 
The parabolic subgroup Q of G has Lie algebra q = [ + u_ and its nilradical u_ is opposite 
to u + , which in turn represents the holomorphic tangent space to D at Zq. According the the 
multiplicity of the noncompact simple root, u + = Ui + U2. The maximal compact subvariety 

Y = Kq(z ) = K(z ) has holomorphic tangent space at zq represented by 112 and has holomorphic 
normal space represented by V = Ui . The group L acts irreducibly on both of them, and those 
representations were derived explicitly in Section [3l The variety Y is a complex flag manifold 
K/ (K fl Q) in its own right, and is the fiber of the basic fibration (|2.8p D — Gq/Lq — » Gq/Kq. 

Fix an irreducible representation r 7 of L. Here 7 is the highest weight, E 7 is the representa- 
tion space, E 7 — > D is the associated homogeneous holomorphic vector bundle, and C(E 7 ) — > D 
is the sheaf of germs of holomorphic sections. 

By 0(E~)|y — > D we mean the pull-back sheaf of C(E 7 ) — > D under Y <^-> D. It is a sheaf 
on D supported on Y . We filter it by order of vanishing in directions transverse to Y: 

(5.6) JF™(E 7 ) = {/ £ C(E 7 )|y J / vanishes to order ±t n in directions transverse to Y\. 

We also need the notation 

Hy — » Y : holomorphic normal bundle to Y in D, 

(5.7) — > Y" : holomorphic conormal bundle to Y in 13 and 

5™(N r ) = 0(S n (Ny)) where S" l (N y ) -> Y is the n t/l symmetric power of N Y Y. 

Then Ny — > Y is the homogeneous holomorphic vector bundle over Y with fiber represented by 

V = Ui, its dual N Y — > Y is the homogeneous holomorphic vector bundle with fiber V* = u_i, 
similarly for the third bundle with fibers S n (V*), and we view <S"(Ny) as a sheaf on D supported 
on Y. Now we have short exact sequences 

(5.8) -> j c "" +1 (E 7 ) -> ^"(E 7 ) -» C»(E 7 | y ® 5"(N r )) ^ 
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of sheaves on D supported in Y . This leads to the long exact sequences 

0^ff°(L>;.F l+1 (E 7 )) ^"(E 7 )) X H (D;O(E 7 \ Y ® S n (N Y ))) ^ 

i/ 1 (^;^ n+1 (E 7 )) ^i/ 1 (^;^ n (E 7 )) if 1 ^; 0(E 7 |y (g> S n (N* Y ))) i» 

(5-9) 

iF^D;^ 1 ^)) ^i? s - 1 (Z3;^"(E 7 )) -^i/ s - 1 (L>;0(E 7 | y ®S'™(N^))) i 
ir(L>;.F l+1 (E 7 )) i/ s (D;^"(E 7 )) i H S (D; C(E 7 |y <g) S ra (N y ))) i» 0. 

where a and & are coefficient morphisms from (I5.8[) . i5 is the coboundary, and s = dim c Y. If a 
sheaf on a locally compact space (such as D) is supported on a closed subspace (such as Y) then 
the inclusion induces a natural isomorphism of cohomologies [GJ Corollary to Lemma 4.9.2]. So 
we can rewrite (|5.9[) as 

-^H (Y;F n+1 (E y )) ^ H°(Y;F n (E 7 )) ^ H {Y;O(E y \ Y ®5"(N y ))) 
i? 1 (F;j !: '™ +1 (E 7 )) Ai^Y;^^)) ff^Y"; 0(E 7 | y ® S"(N y ))) i 

(5-10) 

^-i(y ; jFn+i( E ^)) iJ- s - l (y ; ^»(E T )) i iJ s - 1 (r;C(E 7 |y (g> S n (N y ))) ^ 
^(y ; jm+i(E 7 )) jc-"(E 7 )) ^ H s {Y-0{^\ Y ® S"(N y ))) i 0. 

Note that (|5.10p is an exact sequence of if-modules. 

Remark 5.11 Let n,j ^ 0. Then W{Y; 0(E 7 | y ® S n (H Y ))) = H HY; C>(E 7 | y )) ® 5"(u_i) as 
^-module. If E 7 — > V is a line bundle then Ki acts trivially on the first factor £P(Y; 0(E 7 | y )). 

Proof. The group acts trivially on Y, so its action on 5™(Ny-) factors out of the cohomology. 
Recall that N y — > Y is the ifo _ homogeneous vector bundle based on the Lo-module u_i. If 
E 7 — > Y is a line bundle then i^2 acts trivially on each ffl(Y; 0(E 7 | y )) because it is semisimplc. 



Recall that the positive compact roots are those for which the coefficient of v, as a linear 
combination from = Wg, is or 2. The ones of coefficient are roots of (I, t). The others, 
forming the set A2 of the discussion after (|2.10p . are the complementary compact positive roots. 
They give the holomorphic tangent space of Y. Let p( denote half the sum of the positive 
compact roots (positive roots of ?) . Then the proof of (|2.11ap gives us 

(5.12) (7 + p t , a) < for all a £ A 2 if and only if (7 + p e ,p) < 0. 

If a.\ 6 Ai and 02 € A 2 then a± + 012 is not a root, because it would have coefficient 3 at v. 
Thus (0:1,0(2) ^ 0. That gives us 

Lemma 5.13 If o 2 S A2 then (7 + pt,ot2) ^ (7 + p Sl U2)- In particular if U2 £ A2 then 
(7 + p B , 02) < implies (7 + pt , 02) < 0. T/ius the Go -negativity condition 

(7 + p g , a) < /or all a G A x U A 2 

implies the Kg-negativity condition 

(7 + pt, a) < /or a/Z a G A 2 . 

We are going to need the following fact about tensor products of irreducible finite dimensional 
representations. It appears in [H] as Exercise 12 to Section 24, based on [Kolj . 

Lemma 5.14 Let E lt and E 72 be irreducible L^-modules, where 7$ is the highest weight of E~ fi . 
Then every irreducible summand of E 7l g) E l2 has highest weight of the form 71 + tp for some 
weight (p of E J2 . 
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Now the -Ko-negativity condition gives a vanishing result in (|5. 10|) . as follows, where we take 
(15.12|) into account. 

Theorem 5.15 Suppose that (j + p h p) < 0. Then H j (Y;0(E 1 \ Y ® S n (N Y ))) = whenever 
j =/= s and n ^ 0. 

Proof. Note that E 7 |y (£> S n (N Y ) — > F is the -/^"homogeneous bundle based on the represen- 
tation of L on £L ® ^"(u-i). In view of Lemma [5. 141 that L _m °dule is the sum of irreducibles 
with highest weights of the form 7 + (p where tp is a weight of ^"(u-i). Thus, as a homogeneous 
holomorphic vector bundle, E 7 |y <E> S n (N Y ) has composition series with composition factors of 
the form E 7+Ql _| y an where the on £ A_i. 

Let a G A2. Then (15.12|) shows that (7 + pt, a) < 0. The coefficient of v in a is 2, so a — a; 
cannot be a root. This forces (oti, a) ^ 0. Now (7+pe, a) < forces (7+0:1 + - • -+a n +pt, a) < 0. 
The Bott-Borel-Weil Theorem now tells us that H j (Y; C(E 7 |y ® S" n (N y ))) = for j ^ s. □ 

Corollary 5.16 Suppose that (7 + < 0. Then H^Y; 0(E 7 |y)) = whenever j ^ s. 

Following [S2] and [W4] , with the result [S3] that 7 + p g need only be nonsingular (instead 
of "sufficiently nonsingular"), one has the following vanishing theorem. 

Theorem 5.17 If (7 + p a , a) < whenever a G A x U A 2; i/ien iP(D; C(E 7 )) = /or j ^ s. 

Recall the decomposition of (|2.10p : 7 = 70 + tv* where (70, v) = and t e 1. In view of 
(|2.11a|) . ([2~TlT|) and Theorem [2+21 we reformulate Theorem 15. 171 as follows. 

Theorem 5.18 Let 7 = 70 + tv* as in ()2.10p . // (7 + p B ,^) < and (7 + p s , uP { (v)) < 0, in 
other words if t < —5(70 + Pg, p) an d t < — (70 + Pg, w®(v)), then H^D; C(E 7 )) = for j ^ s. 

Definition 5.19 To facilitate use of these vanishing theorems we will say that E 7 — > D is 
sufficiently negative if {"f+p B , a) < whenever a is a complementary positive root, i.e. whenever 
a £ Ai U A2. This means that E 7 (giK 1 / 2 — > D is negative in the sense of differential or algebraic 
geometry, where K — > D is the canonical line bundle. (} 

In the presence of sufficient negativity Theorem 15.181 trivializes the long exact sequences (I5.9[) 
and (|5.10[) as follows. 

Proposition 5.20 Suppose that E 7 — ► D is sufficiently negative. Then 

H q(Y- _F" +1 (E 7 )) = J ff <? (T;^™(E 7 )) /or ^ q < s, and 
H s {Y;0{E J \ Y ®S n {N* Y )))^H s {Y;T n (E J ))/H s (Y ] F l+1 {E x )). 

Now we may apply the above case-by-case analysis and diagrams to understand what amounts 
to the structure and geometric quantization of the coadjoint elliptic orbits corresponding to the 
particular discrete series of representations we have in mind, the so-called Borel - de Siebenthal 
discrete series. Also, the results above on the filtration are crucial for the construction of the 
cohomology groups carrying these representations; they are the analytic counterparts of the 
Vogan-Zuckerman derived functor modules that are constructed purely algebraically. Thus we 
wish to construct the Borel - de Siebenthal discrete series by direct analysis on orbits, and using 
the above results analyze the ifo~types explicitly (without subscript K denotes the complexified 
group); after this we shall end the paper with some remarks and immediate consequences, and 
treat the (interesting) analytic continuation of this particular discrete series in a later paper. 

As is clear from the above discussion there is some variation in the meaning of "discrete 
series" . Initially the discrete series of Go meant the family of (equivalence classes of) irreducible 
unitary representation 7r of Go that are discrete summands of the left regular representation. 
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This is equivalent to the condition that the matrix coefficients f u ,v(j}) — {u, tt(v)) of it belong to 
L 2 (G) . That is how they are treated in the work of Harish-Chandra, and there the discrete series 
representations are also treated as Harish-Chandra modules. Later one had the construction 
of discrete series representation as the action of Go on cohomology spaces H q (D;E) both as 
nuclear Frechet spaces ([S2 , SW2]) and as Hilbert spaces |W4] , and still later they appeared 
algebraically as Zuckerman derived functor modules. The underlying Harish-Chandra module 
is the same for all these constructions, and we will use the cohomology constructions. 

We first recall some results about the discrete series representations in general. See |Knl[ 
Theorem 9.20] where Harish-Chandra's parameterization is recalled: Here there is given a stan- 
dard root order (which is not the same as we are working with in the diagrams above) of the 
root system as follows: 

(5.21) A+ = {aeA\ (A, a) > 0} 

where A is the root system, and A is the Harish-Chandra parameter for the discrete series 
representation tt\. The Harish-Chandra parameter A £ (it)' satisfies the integrality condition 
that A + p g is analytically integral, in other words that exp(A + p g ) is a well defined character 
on the maximal torus of Kq. It also satisfies the nonsingularity condition that (A, a) ^ for all 
a in A. Two such representations are equivalent if and only if their parameters are conjugate 
under the compact Weyl group We. Thus one could normalize the Harish-Chandra parameter 
by the condition that (A, a) < for all compact positive roots. 

The parameter A of course determines the positive root system A^~ of the standard root 
order (|5.2ip , and conversely to each Weyl chamber of g, modulo the action of Wt, we associate 
a family of discrete series representations. The family we are interested in is in some sense the 
smallest possible kind of discrete series representations of Go- 

Of particular interest is the lowest iio-type contained in the (Harish-Chandra module for) 
tt\ given by its highest weight (in the standard root order (|5.21|) ) 

A = A + p g - 2p { 

in terms of the usual half sums of positive roots. This -Ko~type has multiplicity one, and other 
-Ko-types have highest weights of the form 

A' = A + n a a 

a£A+ 

for integers n a ^ 0. In the general theory of discrete series this statement about the K -types 
only amounts to an inclusion, whereas our results above analyzing the cohomology groups in 
terms of restriction and Taylor expansion in the normal direction (V) gives a concrete list of 
the iio-types. We shall formulate this precisely below. 

Let us first see how these parameters fit with the description in [GWj of the quaternionic 
discrete series n\. They write (3 for the maximal root, but we translate that to our notation of 
p in describing their results. Thus the Harish-Chandra (and infinitesimal character) parameter 
of their 7r* is of the form A = — kp, + p g where the integer k > 2d + 1 and diiriR Go/Kq = Ad. 
We consider the corresponding standard root order A^. Dividing as usual into compact and 
noncompact roots we have p g = pe + P g /e, Pe = P\ + § and p g /i = f/i, where p g /i is half the 
sum of the noncompact positive roots and p\ is half sum of positive roots of l. Similarly for the 
standard root order we have p\ = p\ — ^ and p' B , { = — so the lowest iio - type in the standard 

root order A^ is A = —\p + p g + p' g — 2p' t . That simplifies to A = ~ k ^~ 2 p. This is exactly the 
highest weight for the (k — 1) -dimensional representation of the simple SU (2) factor in K found 
as the lowest K -type by Gross and Wallach. In the following we shall find the analogous lowest 
i^ ~fyP e f° r t ne Borel - de Siebenthal discrete series, and at the same time realize it (and in 
fact all -Kp-types) as cohomology groups on the compact Hermitian symmetric space Y. 
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Now recall the noncompact simple root v from Section [2] As before, v* denote the dual to 
v in the system of fundamental simple weights (|2.9|) . The parabolic subalgebra q of g may also 
be defined by means of v* , and the centralizer of v* is I. Thus the coadjoint orbit Ad*(Go)(^*) 
is our space Gq/Lq and is hbered by Y, Multiples of this v* will define the line bundles we 
shall need, and the representations in the Borel - de Siebenthal discrete series are then the 
cohomology groups in degree s = dime Y with coefficients in the bundle. 

Recall the maximal compact subgroup Kq — K\ x K 2 explicit in the classification of Section 
[3J where the "small" factor K\ corresponds to the component of the simple root system ^ = 

\ \y\) U {— p] that contains {—//}. In the quaternionic case L\ = Sp(l). Now Y = Kq/Lq = 
(Ki x K-i)l{L\ x K2) — K\/L\, Thus, as far as induced representations and cohomology, the 
action of the K2 factor will be rather simple. This we will make explicit below. Also, it is 
important that the factor L\ in Lq contains the center of Lq, and that the action of that center 
on the holomorphic normal space V = Ui is given explicitly in the case by case diagrams of 
Section H 

Let E 7fc — > D be the holomorphic vector bundle induced from the representation of Lq with 
highest weight 7^ = 70 — kv* , k £ N. (As G is simply connected exp(7fe) is the highest weight 
of a representation of Lq). Denote 

(5.22) TT\ k : representation of Go on H S (D, 0(E lk )) where Xk = 7fc + p B — 70 — kv* + p g . 

We will say that the integer k is sufficiently positive if Aj, = 7^ + p g = 70 — kv* + p s is 
sufficiently negative in the sense of Definition 15. 191 and Theorem 12. 121 Using the filtration (|5.6p 
and arguments analogous to those of [GWj we characterize the line bundle valued Borel - de 
Siebenthal discrete series as follows. 

Theorem 5.23 The Borel - de Siebenthal discrete series representations of Gq are the ir\ k of 
(|5.22| for which k € N is sufficiently positive. As a K -module, the underlying Harish- Chandra 
module is 

J2H s (Y,0(E 7k ®S m (V*))), 

and it not only is K -admissible but is K\-admissible. Further, the lowest K -type (correspond- 
ing to m = 0) is given by 

W Xk =H s (Y,0(E lk )), 

and it has multiplicity 1 in 7r\ k . 

Remark. The Lo - modules S m (V*) are not always multiplicity free, though they are multiplicity 
free in many cases. For example for the group of type Dg and m = 6, calculation with the 
computer program LiE produces multiplicities, while there are none for P4. Thus even in the 
scalar case, where E 7n — » D is a line bundle, i.e. when 70 = 0, ir\ k need not be i^o-multiplicity 
free. This is of course in contrast the the iio~multiplicity free property of the line bundle 
holomorphic discrete series. <^> 

Proof. We use the filtration l|5.6p and the exact sequences (|5.9p and (|5.10[) , together with 
the fact that Y is a compact hermitian symmetric space for K\. The action of K2 is part 
of the holomorphically induced representation, and the action of L\ omV and its dual V* is 
given as above. Finally the admissibility can be read off from the i^o _ types directly: each 
H S (Y, 0(E 7fc <S> S m (V*))) is a sum of irreducible representations of Kt, disjoint for different m, 
and the S m (V*) are finite dimensional representations of K2 and also of Lq. 

Consider the parabolic subgroup Q n K — LU^2 of K. Whenever M is a finite dimensional 
(Q fl if)-module, the space of if-finite vectors in the induced representation Indg nK (M) is 
~^2 SG f> V$ ® (Vfi ® M)Q nK . In particular the multiplicity of a -Ko - type 5 is equal to the number 
of times the highest weight vector of M occurs as a highest weight vector for L in V$ . In our case 
the highest weights of M will grow with m in S m (V*), and they are distinguished by the action 
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of the center of Lq. Thus each K\— type only occurs finitely many times, so Tt\ k is K\ -admissible. 
That, of course, implies admissibility for Kq. D 



Remark. We compare our parameter for the lowest K -type with the general description 
mentioned for the scalar quaternionic case. In that scalar quaternionic case the infinitesimal 
character of the representation ir\ k is given by 

A fc = -ku* + p s , p t = pi+ c 2 u* and p 9 / t = c x v* 

for positive constants C\ and c 2 depending only on the root system. Then for the standard root 
order A^" we get 

Pt = pi - c 2 v* and p' n = —cw* 
so that the lowest fCo~type has highest weight 

A = -ku* + p B + p' g - 2p' t = -ku* + 2p e/l 

where p^n = c 2 v* is exactly the shift coming from the square root of the canonical bundle 
K — > Y. Thus this corresponds to the lowest -Ko-type above, viz. W = H S (Y, 0{L-k))- § 

It is an interesting problem to study the structure, including unitarity, of n\ k for smaller 
values of fc, and to relate this to the projective varieties defined by the relative invariants - this 
will be taken up in a sequel to this paper. In particular the ring of regular functions on Lo-orbits 
will be important, as in the paper by Gross and Wallach for the case of the quaternionic discrete 
series. For now we remark as an application of the admissibility above, that branching problems 
will be manageable in a way similar to the case of holomorphic discrete series. This will require 
that the embedding of the smaller group respects the relevant structure, i.e. that the orderings 
are compatible. For example if we want to branch to a symmetric subgroup Hq of Go, then 
the embedding will be compatible provided the symmetry fixes K\ . Namely, we simply use the 
admissibility of the action of K\ , so that admissibility for the branching law to Hq will follow 
for the Borel - de Siebenthal discrete series. In this case a Borel - de Siebenthal discrete series 
will branch as a direct sum of Borel - de Siebenthal discrete series representations. 

Remark. As is evident from the case of indefinite orthogonal groups as in |Kn3j , the question of 
continuation of the discrete series modules is closely connected with the geometry of the relative 
invariants for the holomorphic normal (to the maximal compact subvariety) V. Already the 
case where G is of type Eg and K of type D 8 is an interesting example; here V is of dimension 
64 and admits a relative invariant of degree 8, and the maximal compact subvariety is the 
Grassmannian of 2-planes in 16-space. Let us here be a little more explicit about this example: 
Let 

7o = n 2 & + ... + n 8 £ 8 

and we find 

pt = Uip! + 28^2 + 35^3 + 55^4 + 46^5 + 36^ 6 + 25</> 7 + 13^8 

and 

p s = 46^1 + 68^2 + 91^3 + 135^4 + 110-05 + 84^ 6 + 57^7 + 29tp 8 
so the "sufficient Go-negativity" condition of Theorems 2.12 and 5.18 is, using 

H = 2tpi + 3V> 2 + 4i/>3 + 6?/>4 + 5^5 + 4^6 + 3i/>7 + 2i(j 8 

that bothi < -|(3n 2 + 4n 3 + 6n 4 + 5n 5 + 4n 6 + 3n 7 + 2n 8 ) - ^ and t < -(70 + p 3 , vtf{v)). Now 
the range from Go-sufficiently negative to -Ko-sufficiently negative is indicated by the condition 
(as in Corollary 5.16) t < —^(3ri2 + 4n 3 + 6ri 4 + 5n 5 + 4ri 6 + 3n 7 + 2ri 8 ) + |. 

Hence we see that there is an interval, where the i^o~types still exist as cohomology groups, 
even though the large cohomology group carrying the Go-representation ceases to exist. We 
shall study in more detail what happens here in a sequel to the present paper. 
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